Object detection device and object detection method

ABSTRACT

An image feature map generating unit (3) generates, on the basis of feature amounts extracted from a plurality of images successively captured by a camera (109), an image feature map which is an estimated distribution of the object likelihood on each of the images. An object detecting unit (4) detects an object on the basis of the image feature map generated by the image feature map generating unit (3).

TECHNICAL FIELD

The present invention relates to an object detection device and anobject detection method for detecting an object from a video captured bya camera.

BACKGROUND ART

As a technique for detecting an object from a video captured by acamera, for example, an object detection device described in PatentLiterature 1 can be mentioned. In this device optical flows of a screenassumed in a standard environment in which no object is present isgenerated, also optical flows based on a video actually captured by acamera is generated, and an object is detected on the basis ofdifferences between both.

Note that an optical flow is information in which the amount of movementof the same object associated between successive frame images capturedat different times is represented by a vector.

CITATION LIST Patent Literatures

Patent Literature 1: JP 2007-334859 A

SUMMARY OF INVENTION Technical Problem

In an optical flow, because the number of pixels of an object in acaptured image decreases as the object is located farther from thecamera, an absolute value of a vector representing the amount ofmovement of the object becomes smaller.

Therefore, when an optical flow based on images of an object located farfrom the camera is used, a difference from an optical flow obtained froman image in which no object is present cannot be obtained accurately,and the object cannot be detected with high accuracy. That is, there isa problem in the object detection device described in Patent Literature1 that an object located far from the camera cannot be accuratelydetected.

The invention is to solve the above problem, and it is an object of thepresent invention to provide an object detection device and an objectdetection method capable of accurately detecting an object within arange from the vicinity of a camera to a distant location.

Solution to Problem

An object detection device according to the present invention includesan image feature map generating unit and an object detecting unit.

In this configuration, the image feature map generating unit generates,on the basis of feature amounts extracted from a plurality of imagessuccessively captured by a camera, an image feature map representing anestimated distribution of the object likelihood on each of the images.The object detecting unit detects an object on the basis of the imagefeature map generated by the image feature map generating unit.

Advantageous Effects of Invention

According to the present invention, since an object is detected on thebasis of an estimated distribution of object likelihood on an image, anobject can be accurately detected within a range from the vicinity ofthe camera to a distant location, the range being captured by thecamera.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram illustrating a functional configuration of anobject detection device according to a first embodiment of the presentinvention.

FIG. 2 is a block diagram illustrating a hardware configuration of theobject detection device according to the first embodiment.

FIG. 3 is a flowchart illustrating the operation of the object detectiondevice according to the first embodiment.

FIG. 4 is a flowchart illustrating a specific example of processing insteps ST4 and ST5 in FIG. 3.

FIG. 5 is a flowchart illustrating generation processing of a saliencymap.

FIG. 6 is a block diagram illustrating a functional configuration of anobject detection device according to a second embodiment of the presentinvention.

FIG. 7 is a block diagram illustrating a hardware configuration of theobject detection device according to the second embodiment.

FIG. 8 is a flowchart illustrating the operation of the object detectiondevice according to the second embodiment.

FIG. 9 is a flowchart illustrating a specific example of processing insteps ST5 c and ST7 c in FIG. 8.

FIG. 10 is a graph illustrating a relationship between the distance froma vehicle to a moving body and the coincidence rate.

DESCRIPTION OF EMBODIMENTS

To describe the present invention further in detail, embodiments forcarrying out the invention will be described below with reference to theaccompanying drawings.

First Embodiment

FIG. 1 is a block diagram illustrating a functional configuration of anobject detection device 1 according to a first embodiment of the presentinvention. The object detection device 1 detects an object from a videocaptured by a camera. An object to be detected may be a moving body suchas an individual or a vehicle or a stationary object such as a sign.

As illustrated in FIG. 1, the object detection device 1 includes a videocapturing unit 2, an image feature map generating unit 3, an objectdetecting unit 4, and an object recognizing unit 5.

The video capturing unit 2 acquires video data captured by the camera.The video data is an image sequence of a plurality of imagessuccessively captured by the camera, and individual images arranged in atime series are frame images.

Note that the camera may be a fixed camera fixedly provided at apredetermined position, or may be a camera mounted on a moving body suchas a vehicle.

The image feature map generating unit 3 generates image feature maps onthe basis of feature amounts extracted from the video data captured bythe camera. An image feature map is a map representing an estimateddistribution of object likelihood on an image. The object likelihoodmeans, for example, the degree of being an object or a target of sometype.

For example, the image feature map generating unit 3 generates an imagepyramid including a plurality of images having different image sizesobtained by gradually reducing a frame image. Subsequently, the imagefeature map generating unit 3 extracts, from each of the images in theimage pyramid, feature amounts of respective image features on thecorresponding image, and maps the extracted feature amounts to atwo-dimensional coordinate system. This map is an image feature mapillustrating an estimated distribution of the object likelihood in thecorresponding image.

The object detecting unit 4 detects an object on the basis of the imagefeature map generated by the image feature map generating unit 3. Forexample, the object detecting unit 4 detects an object on the imageusing the image feature map. The object recognizing unit 5 recognizesthe object detected by the object detecting unit 4. For example, theobject recognizing unit 5 recognizes an attribute of the object on thebasis of the shape or the like of the object detected by the objectdetecting unit 4.

Although in FIG. 1 the case where the object detection device 1 includesthe video capturing unit 2 has been described, the video capturing unit2 may be included in the camera itself.

Moreover, the object recognizing unit 5 may not be included in theobject detection device 1 but may be included in an external deviceconnected subsequently to the object detection device 1.

That is, the object detection device 1 is only required to include atleast the image feature map generating unit 3 and the object detectingunit 4.

FIG. 2 is a block diagram illustrating a hardware configuration of theobject detection device 1. In FIG. 2, the video capturing unit 2illustrated in FIG. 1 fetches an image sequence captured by a camera 109via a camera interface 106 and stores the image sequence in a data readonly memory (ROM) 101. The image feature map generating unit 3illustrated in FIG. 1 generates image feature maps using the imagesequence stored in the data ROM 101 and stores the generated imagefeature maps in a random access memory (RAM) 103.

The object detecting unit 4 illustrated in FIG. 1 detects an object byusing an estimated distribution corresponding to each of the pluralityof images having different image sizes stored in the RAM 103. Thedetection result of the object is stored in an external memory 107 via adisk controller 104, or displayed on a display device 108 via a displaycontroller 105.

Moreover, the object recognizing unit 5 illustrated in FIG. 1 recognizesan attribute of the object detected by the object detecting unit 4. Anattribute of an object includes, for example, a type such as a vehicle,an individual, or a two-wheeler.

Note that the recognition result of the object is either stored in theexternal memory 107 via the disk controller 104 or displayed on thedisplay device 108 via the display controller 105.

Note that the disk controller 104, the display controller 105, thecamera interface 106, the external memory 107, the display device 108,and the camera 109 may not be included in the object detection device 1.That is, these devices may be provided separately from the objectdetection device 1, and may be included in an external device capable ofreceiving and outputting data from and to the object detection device 1.

Note that the functions of the image feature map generating unit 3 andthe object detecting unit 4 in the object detection device 1 areimplemented by a processing circuit. That is, the object detectiondevice 1 includes a processing circuit for generating image feature mapson the basis of feature amounts extracted from a plurality of imagescaptured successively by the camera and detecting an object on the basisof the image feature maps. The processing circuit may be dedicatedhardware or a central processing unit (CPU) 100 that executes a programstored in a program ROM 102.

In the case where the processing circuit is the hardware, the processingcircuit corresponds to, for example, a single circuit, a compositecircuit, a programmed processor, a parallel programmed processor, anapplication specific integrated circuit (AISC), a field-programmablegate array (FPGA), or a combination thereof.

In addition, each of the functions of the image feature map generatingunit 3 and the object detecting unit 4 may be implemented by aprocessing circuit, or the functions may be implemented by a singleprocessing circuit in an integrated manner.

In the case where the processing circuit is the CPU 100, the functionsof the image feature map generating unit 3 and the object detecting unit4 are implemented by software, firmware, or a combination of softwareand firmware. The software and the firmware are described as programsand stored in the program ROM 102. The CPU 100 reads and executes theprograms stored in the program ROM 102 and thereby implements thefunctions.

That is, the object detection device 1 includes a memory for storingprograms which result in, when executed by the processing circuit,execution of the step of generating an image feature map and the step ofdetecting an object on the basis of the image feature map.

These programs also cause a computer to execute a procedure or a methodof each of the image feature map generating unit 3 and the objectdetecting unit 4. The memory may include, for example, a nonvolatile orvolatile semiconductor memory such as a RAM, a ROM, a flash memory, anerasable programmable ROM (EPROM), or an electrically EPROM (EEPROM), amagnetic disc, a flexible disc, an optical disc, a compact disc, a minidisc, or a digital versatile disk (DVD).

Furthermore, some of the functions of the image feature map generatingunit 3 and the object detecting unit 4 may be implemented by dedicatedhardware, and the other may be implemented by software or firmware.

For example, the function of the image feature map generating unit 3 isimplemented by a dedicated hardware processing circuit while thefunction of the object detecting unit 4 is implemented by execution ofthe programs stored in the program ROM 102 by the CPU 100.

In this manner, the processing circuit can implement the functionsdescribed above by hardware, software, firmware, or a combinationthereof.

Next, the operation will be described.

FIG. 3 is a flowchart illustrating the operation of the object detectiondevice 1 and a series of processes until an object is detected.

The video capturing unit 2 fetches video data captured by the camera 109(step ST1). If the shooting by the camera 109 is finished (step ST2:YES), the series of processes illustrated in FIG. 3 is terminated. Thatis, fetching of video data by the video capturing unit 2 is continueduntil the shooting by the camera 109 is completed. For example, in thecase where the object detection device 1 is a vehicle detection deviceusing an in-vehicle camera, shooting the outside of the vehicle by thein-vehicle camera is continued while the vehicle is traveling.

When shooting by the camera 109 is not finished (step ST2: NO), theimage feature map generating unit 3 receives an image sequence fetchedby the video capturing unit 2, reduces a frame image in the imagesequence gradually to generate an image pyramid (step ST3). Theprocessing of step ST3 is repeated as many times as the number of frameimages in the image sequence.

Next, the image feature map generating unit 3 extracts feature amountsfrom each of the plurality of images having different image sizes in theimage pyramid and generates an image feature map for each of the images(step ST4). The processing of step ST4 is repeated as many times as thenumber of image pyramids.

The object detecting unit 4 integrates the image feature maps obtainedfor the respective images in the image pyramid, as an estimation resultfor one frame image, and detects an object on the basis of a result ofthe integration (step ST5). The detection result of an object obtainedin this manner is output from the object detecting unit 4 to the objectrecognizing unit 5, and the object recognizing unit 5 recognizes anattribute or the like of the object.

Here, detection of an object based on an image feature map will bedescribed in detail.

FIG. 4 is a flowchart illustrating a specific example of processing insteps ST4 and ST5 in FIG. 3.

First, the image feature map generating unit 3 acquires a frame image inthe image sequence fetched by the video capturing unit 2 (step ST1 a),and acquires the same frame image from the image sequence and invertsthe frame image (step ST2 a). Here, to invert means to invert thebrightness in the frame image. That is, a dark part of the image isconverted to a bright part, and a bright part is converted to a darkpart.

Next, the image feature map generating unit 3 generates an image pyramidby reducing the acquired frame image gradually, and further generates animage pyramid by reducing the inverted frame image gradually.Subsequently, the image feature map generating unit 3 generates asaliency map from the image pyramid of the frame image not inverted(step ST3 a), and generates a saliency map from the image pyramid of theinverted frame image (Step ST4 a).

Here, a saliency map is a map representing a salient region that isdifferent from its surrounding region on an image. The above surroundingregion is a region at which humans are likely to gaze on the image onthe basis of a human visual model. Here, the salient region correspondsto the estimated distribution of object likelihood, and the saliency mapis a specific example of the image feature map.

Next, the object detecting unit 4 integrates saliency maps obtained forthe respective images in the image pyramid, as an estimation result onone frame image. This processing is performed both on the saliency mapsobtained from the image pyramid for the frame image not inverted and thesaliency maps obtained from the image pyramid for the inverted frameimage, which are further integrated.

Subsequently, the object detecting unit 4 compares image features of thesaliency map with a threshold value related to an image feature todetermine whether there is a region having a feature amount larger thanthe threshold value (step ST5 a).

Here, if there is no region having a feature amount larger than thethreshold value in the saliency map (step ST5 a: NO), it is determinedthat no object has been detected, and the processing is terminated.

If there is a region having a feature amount larger than the thresholdvalue in the saliency map (step ST5 a: YES), the object detecting unit 4detects this region as a region having a detection target object therein(step ST6 a). Thereafter, the object detecting unit 4 groups the regionsextracted in the above manner, and outputs them to the objectrecognizing unit 5 as a detection region of the object.

Here, the aforementioned generation processing of the saliency map willbe described in detail.

FIG. 5 is a flowchart illustrating generation processing of a saliencymap.

First, the image feature map generating unit 3 converts an image to beprocessed into a Lab space designed by approximation to perceptionlevels of colors by humans (step ST1 b).

Subsequently, the image feature map generating unit 3 calculates anaverage color of the image converted into the Lab space (step ST2 b).This average color is a representative color of this image.

Next, the image feature map generating unit 3 applies a difference ofGaussian (DoG) filter to the image converted into the Lab space (stepST3 b). As a result, Gaussian filters having different scales in the DoGfilter are applied to each pixel value of the image, and differencestherebetween are obtained.

In the human perception system, retinal cells are known to perceivelight intensity and edge directivity from the difference between thecenter and its surroundings. The DoG filter imitates such operation ofretinal cells by image processing.

Out of the Gaussian filters in the DoG filter, application of the onewith a smaller scale results in an image having a high resolution, andapplication of the one with a larger scale results in a blurred imagehaving a low resolution. Utilizing differences in corresponding pixelvalues between both images means to utilize differences in pixel valuesbetween a pixel of interest and its surrounding pixels, which makes itpossible to obtain an image having a larger change as compared to thesurrounding pixels.

Subsequently, the image feature map generating unit 3 calculates adifference between the color of the image to which the DoG filter isapplied and the average color calculated in step ST2 b (step ST4 b). Asa result, a salient region having a large deviation width from theaverage color is left, thereby enabling removal of the representativecolor of the peripheral region of this region. In this manner, in stepST4 b an overall salient part is obtained for the entire image.

Note that it suffices to obtain images having different resolutions inthe processing of the DoG filter described above, and thus without beinglimited to the Gaussian filter, it is also possible to performprocessing of resizing an image to images having different sizes andthen restoring them to images having the original size.

Next, the image feature map generating unit 3 applies an adaptivebinarization filter to the image processed in step ST4 b (step ST5 b).By the adaptive binarization filter, it is not that the entire image isbinarized by using a threshold value, but by using a threshold valuedetermined for each of the pixels in the image, a corresponding one ofthe pixels is filtered. As a result, each of the pixels in the image iscompared with the corresponding threshold value for each of the pixels,and a pixel whose pixel value is larger than the corresponding thresholdvalue is extracted.

Note that the threshold value for each pixel is determined on the basisof brightness information in the vicinity of the target pixel. In thecase where the vicinity of the target pixel is bright, a high thresholdvalue is set, and in the case where the vicinity of the target pixel isdark, a low threshold value is set.

Next, the image feature map generating unit 3 applies a Gaussian filterto a group of salient pixels extracted in step ST5 b, thereby obtaininga region of the group of salient pixels as a map (step ST6 b). Then, theimage feature map generating unit 3 binarizes the map (step ST7 b). Theobject detecting unit 4 detects an object on the basis of the mapbinarized in this manner.

Note that, in step ST5 b, more local salient pixels are obtained bynarrowing down the salient part obtained in step ST4 b. This enablesidentification of an edge component that is robust to a local brightnesschange contained in pixels and has a pattern different from those of thesurroundings.

In addition, since in FIG. 4 the image not inverted in step ST1 a isused as well as the image inverted in step ST2 a, it is possible toextract, in similar manners, a dark salient point in the case where thesurrounding region of the pixel of interest is bright as well as abright salient point in the case where the surrounding region of thepixel of interest is dark.

As described above, the object detection device 1 according to the firstembodiment includes the image feature map generating unit 3 and theobject detecting unit 4. In this configuration, the image feature mapgenerating unit 3 generates, on the basis of feature amounts extractedfrom a plurality of images successively captured by the camera 109, animage feature map representing an estimated distribution of the objectlikelihood on each of the images. The object detecting unit 4 detects anobject on the basis of the image feature map generated by the imagefeature map generating unit 3.

With this configuration, since the object is detected on the basis ofthe estimated distribution of object likelihood on the correspondingimage, the object can be accurately detected within a range from thevicinity of the camera 109 to a distant location, the range beingcaptured by the camera 109.

Since the object detection device 1 detects a region of an object thatis noticeable by human eyes in an image, it is effective in detection ofsigns, individuals, defects, or vehicles, for example.

In many cases, characters on signs are written in a color different fromthat of the background part in order to enhance visibility. Therefore,the character part is easily detected by the object detection device 1as a salient region different from the background part.

Moreover, the object detection device 1 does not detect a pattern(texture) of the background, but detects a salient region different fromthe surroundings. Therefore, in detection of individuals, unlessclothing of an individual blends into the background, the clothing ofthe individual is detected as a salient region different from thebackground.

Furthermore, by using the object detection device 1, for example it ispossible to detect parts on a conveyor line in a factory to measure thenumber of the parts, and to discriminate results of forming the partsfrom shapes of the parts recognized by the object recognizing unit 5.

In the case of detecting cracks or the like of a structure, in therelated art a repair mark of the structure or the original pattern orthe like of the structure is also detected, and thus processing ofdistinguishing these from cracks is necessary.

In contrast, in the object detection device 1, since a pattern includedin the background is not detected, cracks of a structure can be easilydetected.

Furthermore, the object detection device 1 is capable of detecting othervehicles on the basis of video data from an in-vehicle camera such as acamera used in a drive recorder. In this case, a region of an objecthaving a color and a shape different from those of the background regionin the image is detected as a region of a vehicle.

Second Embodiment

FIG. 6 is a block diagram illustrating a functional configuration of anobject detection device 1A according to a second embodiment of thepresent invention. In FIG. 6, the same components as those in FIG. 1 aredenoted with the same symbols and descriptions thereof are omitted.

In the object detection device 1A, object detection based on opticalflows and object detection based on an image feature map are performeddepending on the reliability of the optical flows.

As illustrated in FIG. 6, the object detection device 1 A includes avideo capturing unit 2, an image feature map generating unit 3, anobject detecting unit 4A, an object recognizing unit 5, an optical flowcalculating unit 6, a reliability calculating unit 7, and a noiseremoving unit 8.

The optical flow calculating unit 6 calculates optical flows betweenframe images of video data captured by a camera.

An optical flow is information in which the amount of movement of thesame object associated between frame images is represented by a vector,which is calculated for each pixel.

Note that, in optical flows, not only movement information in the timedirection of an object but also spatial continuity is considered, whichenables vector notation reflecting the shape of the object as a feature.

The reliability calculating unit 7 calculates the reliability of theoptical flows. For example, the magnitude of a vector indicating theamount of movement of the object between the frame images, that is, ascalar value is calculated as the reliability. An object located farfrom the camera has a smaller scalar value because an apparent motion onan image captured by the camera is small.

The noise removing unit 8 removes optical flows in a direction along amoving direction of the camera out of the optical flows, as noise. Forexample, in a case where a camera is mounted on a vehicle, optical flowsobtained from images captured by the camera are predominantly thoseobserved in a traveling direction of the vehicle. The optical flows inthis direction are included in the background region of the object, andoptical flows in a direction not equivalent to this direction can beconsidered to be included in the foreground, that is, a region in whichthe object is present. Therefore, the noise removing unit 8 removes theoptical flows included in this background region.

The object detecting unit 4A performs object detection based on opticalflows and object detection based on an image feature map depending onthe reliability of the optical flows. For example, out of regions on animage captured by the camera, the object detecting unit 4A performsobject detection based on optical flows in a region in which scalarvalues of the optical flows are higher than a threshold value, and in aregion in which scalar values are less than or equal to the thresholdvalue, performs object detection based on an image feature map.

Note that, as described earlier, the object detection based on opticalflows is to detect an object on the basis of differences between opticalflows of a screen assumed in a standard environment in which no objectis present and optical flows based on a video actually captured by thecamera.

The object detection based on an image feature map is as described inthe first embodiment.

Although in FIG. 6 the case where the object detection device 1Aincludes the video capturing unit 2 has been described, the videocapturing unit 2 may be included in the camera itself.

Moreover, the object recognizing unit 5 may not be included in theobject detection device 1A but may be included in an external deviceconnected subsequently to the object detection device 1A.

Furthermore, the noise removing unit 8 may be one of functions of theobject detecting unit 4A.

That is, the object detection device 1A is only required to include atleast the image feature map generating unit 3, the object detecting unit4A, the optical flow calculating unit 6, and the reliability calculatingunit 7.

FIG. 7 is a block diagram illustrating a hardware configuration of theobject detection device 1A.

In FIG. 7, the video capturing unit 2 illustrated in FIG. 6 fetches animage sequence captured by a camera 209 via a camera interface 206 andstores the image sequence in a data ROM 201.

The optical flow calculating unit 6 illustrated in FIG. 6 develops theimage sequence stored in the data ROM 201 in a RAM 203 and calculates amotion vector of an object between frame images for each pixel. Thereliability calculating unit 7 illustrated in FIG. 6 calculates anabsolute value (scalar value) of the vector which is an optical flowdeveloped in the RAM 203.

For each of the images developed in the RAM 203, the noise removing unit8 illustrated in FIG. 6 removes optical flows included in the backgroundregion, and holds optical flows included in a region of an object thatis the foreground in the RAM 203. Note that in the RAM 203 a pluralityof optical flows included in the region of the object is held in a timeseries, and thus it is guaranteed that the direction of the opticalflows is stable.

The image feature map generating unit 3 illustrated in FIG. 6 generatesimage feature maps using the image sequence stored in the data ROM 201and stores the generated image feature maps in the RAM 203.

The object detecting unit 4A illustrated in FIG. 6 performs objectdetection based on the optical flows stored in the RAM 203 and objectdetection based on the image feature maps.

Moreover, the object recognizing unit 5 illustrated in FIG. 6 recognizesan attribute of an object detected by the object detecting unit 4A. Anattribute of an object includes, for example, a type such as a vehicle,an individual, or a two-wheeler.

Note that the detection result of the object is either stored in anexternal memory 207 via a disk controller 204 or displayed on a displaydevice 208 via a display controller 205.

The detection result of the object by the object detecting unit 4A andthe recognition result of the object by the object recognizing unit 5are output to a vehicle body controlling unit 210. Here, the vehiclebody controlling unit 210 is a device provided subsequently to theobject recognizing unit 5 in FIG. 6 and controls a brake 211 and asteering 212.

For example, when avoiding a collision between the object detected bythe object detection device 1A and the vehicle, the vehicle bodycontrolling unit 210 controls the brake 211 and the steering 212 toperform driving operation for avoiding the collision. Furthermore, thevehicle body controlling unit 210 determines the optimum drivingbehavior in relation between the object and the vehicle from theattribute of the object recognized by the object recognizing unit 5 andcontrols the brake 211 and the steering 212 to perform the drivingbehavior.

Note that the disk controller 204, the display controller 205, thecamera interface 206, the external memory 207, the display device 208,and the camera 209 may not be included in the object detection device1A. That is, these devices may be provided separately from the objectdetection device 1A, and may be included in an external device capableof receiving and outputting data from and to the object detection device1A.

Note that the functions of the image feature map generating unit 3, theobject detecting unit 4A, the optical flow calculating unit 6, and thereliability calculating unit 7 in the object detection device 1A areimplemented by a processing circuit.

That is, the object detection device 1A includes a processing circuitfor performing operations of the functions described above. Theprocessing circuit may be dedicated hardware or a CPU 200 that executesa program stored in a program ROM 202.

In the case where the processing circuit is hardware, the processingcircuit corresponds to, for example, a single circuit, a compositecircuit, a programmed processor, a parallel programmed processor, anAISC, an FPGA, or a combination thereof.

In addition, each of the functions of the image feature map generatingunit 3, the object detecting unit 4A, the optical flow calculating unit6, and the reliability calculating unit 7 may be implemented by aprocessing circuit, or the functions may be implemented by a singleprocessing circuit in an integrated manner.

In the case where the processing circuit is a CPU 200, the functions ofthe image feature map generating unit 3, the object detecting unit 4A,the optical flow calculating unit 6, and the reliability calculatingunit 7 are implemented by software, firmware, or a combination ofsoftware and firmware.

The software and the firmware are described as programs and stored inthe program ROM 202. The CPU 200 reads and executes the programs storedin the program ROM 202 and thereby implements the functions. In otherwords, the object detection device 1A includes a memory for storingprograms which result in execution of operations of the functions. Theseprograms also cause a computer to execute a procedure or a method ofeach of the image feature map generating unit 3, the object detectingunit 4A, the optical flow calculating unit 6, and the reliabilitycalculating unit 7.

Like in the first embodiment, the memory may be, for example, anonvolatile or volatile semiconductor memory such as a RAM, a ROM, aflash memory, an EPROM, or an EEPROM, a magnetic disk, a flexible disk,an optical disk, a compact disk, a mini disk, a DVD, or the like.

Furthermore, some of the functions of the image feature map generatingunit 3, the object detecting unit 4A, the optical flow calculating unit6, and the reliability calculating unit 7 may be implemented bydedicated hardware, and the others may be implemented by software orfirmware. For example, the function of the image feature map generatingunit 3 is implemented by a dedicated hardware processing circuit whilethe functions of the object detecting unit 4A, the optical flowcalculating unit 6, and the reliability calculating unit 7 areimplemented by execution of the programs stored in the program ROM 202by the CPU 200. In this manner, the processing circuit can implement thefunctions described above by hardware, software, firmware, or acombination thereof.

Next, the operation will be described.

FIG. 8 is a flowchart illustrating the operation of the object detectiondevice 1 A and a series of processes until an object is detected.

First, the video capturing unit 2 fetches video data captured by thecamera 209 (step ST1 c). Here, it is assumed that the camera 209 is in amobile state. This state means, for example, that the camera 209 is anin-vehicle camera and that the camera 209 can move together with thevehicle. Note that the camera 209 may not be moving while capturing avideo.

If the shooting by the camera 209 is finished (step ST2 c: YES), theseries of processes illustrated in FIG. 8 is terminated. That is,fetching of video data by the video capturing unit 2 is continued untilthe shooting by the camera 209 is completed. If shooting by the camera209 is not finished (step ST2 c: NO), the optical flow calculating unit6 calculates an optical flow for each pixel between frame images in animage sequence fetched by the video capturing unit 2 (step ST3 c). Forexample, dense optical flows are calculated.

Next, the reliability calculating unit 7 calculates a scalar value of anoptical flow as reliability.

An object located far from the camera 209 has a smaller scalar valuesince an apparent motion on an image captured by the camera 209 issmall. In addition, a scalar value of an optical flow calculated from anobject moving at an equal speed with the vehicle provided with thecamera 209 is very small.

The reliability calculating unit 7 compares the scalar value of theoptical flow with a threshold value and thereby determines whether thescalar value is larger than the threshold value (step ST4 c).

As the threshold value, by a discrimination analysis method usingabsolute values of motion vectors of optical flows, a value isadaptively determined that makes it possible to appropriately separateregions on the image into a region in which a moving body is present andthe other regions.

If the scalar value is larger than the threshold value (step ST4 c:YES), the reliability calculating unit 7 determines that among regionsin the image, the reliability of optical flows in a region from whichthe optical flow of this scalar value has been obtained is high. Thisdetermination result is notified from the reliability calculating unit 7to the noise removing unit 8. Upon receiving this notification, from theoptical flows in the region having the high reliability of the opticalflows, the noise removing unit 8 removes optical flows of the backgroundregion, as noise (step ST5 c).

On the other hand, if the scalar value is less than or equal to thethreshold value (step ST4 c: NO), the reliability calculating unit 7determines that among regions in the image, the reliability of opticalflows in a region from which the optical flow of this scalar value hasbeen obtained is low. The determination result is notified from thereliability calculating unit 7 to the image feature map generating unit3. The image feature map generating unit 3 generates an image featuremap in a similar manner to the processing in FIG. 4 and FIG. 5 describedin the first embodiment (step ST6 c)

Out of regions on the image, the object detecting unit 4A performs, in aregion in which the reliability of optical flows is high, objectdetection based on optical flows, and, in a region in which thereliability of optical flows is low, performs object detection based onan image feature map (step ST7 c).

The detection result of an object obtained in this manner is output fromthe object detecting unit 4A to the object recognizing unit 5, and theobject recognizing unit 5 recognizes an attribute or the like of theobject.

Here, the noise removal processing by the noise removing unit 8 will bedescribed in detail.

FIG. 9 is a flowchart illustrating a specific example of processing insteps ST5 c and ST7 c in FIG. 8.

First, the noise removing unit 8 separates a background region in animage on the basis of direction components of optical flows (step ST1d).

For example, by using the k-means method, the noise removing unit 8separates a frame image into a region including optical flows in thedominant direction and a region including optical flows in a directionnot equivalent thereto. In this embodiment, a region including opticalflows in the dominant direction is regarded as the background region,and a region including optical flows in a direction not equivalent tothe dominant direction is regarded as the foreground region.

Next, the noise removing unit 8 removes optical flows included in thebackground region (step ST2 d). For example, the noise removing unit 8removes the optical flows included in the background region on the basisof a dynamic background subtraction method extended in a time series.Note that the dynamic background subtraction method is a method forobtaining the foreground region that is not included in the backgroundregion by dynamically generating and updating a background model fromframe images aligned in a time series.

The object detecting unit 4A confirms in a time series whether theoptical flows included in the frame image from which the noise has beenremoved by the noise removing unit 8 are stably oriented in the samedirection (step ST3 d). For example, the object detecting unit 4Aestimates the position of the foreground region in a next frame image,by utilizing the direction of the optical flows from a frame image whichis the preceding frame image and from which the noise of the foregroundregion has been removed. By putting this estimation result and theactual next frame image together, the region in which the object ispresent is estimated in a time series. The object detecting unit 4Aperforms this correction processing for a predetermined number ofrepetitions.

Next, the object detecting unit 4A determines whether an absolute valueof a vector of an optical flow in the foreground region, in which thetime series position data has been corrected in step ST3 d, is largerthan a threshold value (step ST4 d). Here, if the absolute value of thevector is less than or equal to the threshold value (step ST4 d: NO),the object detecting unit 4A determines that the foreground region isnot a region of a moving body and terminates the processing.

On the other hand, if the absolute value of the vector is larger thanthe threshold value (step ST4 d: YES), the object detecting unit 4Adetects the foreground region as a region in which a moving body ispresent (step ST5 d).

Thereafter, the object detecting unit 4A groups the regions extracted inthe above manner and outputs them, as a detection region of the movingbody, to the object recognizing unit 5.

FIG. 10 is a graph illustrating a relationship between the distance froma vehicle to a moving body and the coincidence rate. In FIG. 10, thehorizontal axis represents the distance from the vehicle to the movingbody, and the vertical axis represents the coincidence rate between thecorrect position of the moving body and the position of a detectionresult.

Results denoted by symbols a1 to a3 are those of a conventional objectdetection device described in the following Reference Literature 1, andthe other results are those of the object detection device 1A.

(Reference Literature 1) Norio Hashiguchi, Masatoshi Touno, DaisukeUeno, Yasuhiko Nakano, “Sensing Technology Supporting ConvergenceService”, FUJITSU Technical Report, Vol. 64, pp. 74-80, 2013.

Because the object detection device described in the above referenceliterature largely depends on a calculation result of optical flows, asillustrated by symbols a1 to a3, the object detection device can onlydetect an object in the vicinity of a vehicle and cannot cope with anobject that is far from the vehicle.

On the other hand, because the object detection device 1A detects adistant object on the basis of a saliency map not dependent on themovement of an object, it is possible to stably detect even an object100 m or more away from the vehicle.

Note that, when the video data fetched by the video capturing unit 2 iscompressed, the optical flow calculating unit 6 may calculate an opticalflow using the compressed information.

Among compression methods, there is a method for performing motionprediction of video data by using preceding and succeeding frame images,and using this method enables extraction of a motion region having asimilar gradient direction for each block. By using this motioninformation, only a moving object can be extracted. In this case, sincethe motion information is included in the compressed video data, thereis no need to analyze the video to newly calculate optical flows. As aresult, the calculation load can be reduced.

As described above, the object detection device 1A according to thesecond embodiment includes the optical flow calculating unit 6 and thereliability calculating unit 7 in addition to the configuration of theobject detection device 1 according to the first embodiment.

In this configuration, the object detecting unit 4A performs objectdetection based on optical flows and object detection based on an imagefeature map depending on the reliability calculated by the reliabilitycalculating unit 7. For example, in the case where an object is far fromthe camera 209, the reliability of the optical flows is low and objectdetection based on an image feature map is performed, and in the casewhere an object is in the vicinity of the camera 209, the reliability ishigh and object detection based on optical flows is performed.

As a result, the object can be accurately detected within a range fromthe vicinity of the camera 209 to a distant location.

Furthermore, the object detection device 1A according to the secondembodiment includes the noise removing unit 8. Out of regions on animage, the object detecting unit 4A determines a region in which anoptical flow in a direction not equivalent to that of an optical flowremoved by the noise removing unit 8 is obtained, as a region in whichan object is present.

With this configuration, the region in which the object is present canbe detected accurately.

Note that, within the scope of the present invention, the presentinvention may include a flexible combination of the embodiments, amodification of any component of the embodiments, or omission of anycomponent in the embodiments.

INDUSTRIAL APPLICABILITY

Since the object detection device according to the present invention iscapable of accurately detecting an object within a range from thevicinity of a camera to a distant location, the object detection deviceis suitable for detection of a vehicle, an individual, and the like, forexample.

REFERENCE SIGNS LIST

1, 1A: Object detection device, 2: Video capturing unit, 3: Imagefeature map generating unit, 4, 4A: Object detecting unit, 5: Objectrecognizing unit, 6: Optical flow calculating unit, 7: Reliabilitycalculating unit, 8: Noise removing unit, 100, 200: CPU, 101, 201: DataROM, 102, 202: Program ROM, 103, 203: RAM, 104, 204: Disk controller,105, 205: Display controller, 106, 206: Camera interface, 107, 207:External memory, 108, 208: Display device, 109, 209: Camera, 210:Vehicle body controlling unit, 211: Brake, 212: Steering.

1-5. (canceled)
 6. An object detection device, comprising: a processorto execute a program; and a memory to store the program which, whenexecuted by the processor, performs processes of, acquiring, from aplurality of frame images arranged in a time series in an image sequencesuccessively captured by a camera, a pair of same frame images for eachof the plurality of frame images and inverting one of the same frameimages, generating image pyramids each of which includes images obtainedby gradually reducing a corresponding one of the same frame images ofthe pair, extracting, from an image of a salient part extracted fromeach of the images in each of the image pyramids, a group of pixels eachof whose pixel value is larger than a corresponding threshold value, thethreshold value being set for each of the pixels depending on brightnessinformation in a vicinity of a target pixel, generating a map byintegrating the extracted group of pixels, binarizing and thenintegrating the map generated for each of the images in each of theimage pyramids which correspond to the same frame images of the pair,and thereby generating, for each of the plurality of frame images, animage feature map representing an estimated distribution of objectlikelihood on a corresponding one of the plurality of frame images; anddetecting an object on a basis of the image feature map generated. 7.The object detection device according to claim 6, the processes furthercomprising: calculating optical flows between the plurality of frameimages captured successively by the camera; and calculating reliabilityof the optical flows calculated, wherein object detection based on atleast one of the optical flows and object detection based on the imagefeature map are performed depending on the reliability calculated
 8. Theobject detection device according to claim 7, wherein, out of regions oneach of the plurality of frame images, in a region in which reliabilityof a first optical flow is higher than a threshold value, objectdetection based on the first optical flow is performed, and in a regionin which reliability of a second optical flow is less than or equal tothe threshold value, object detection based on the image feature map isperformed, the first and second optical flows being included in theoptical flows.
 9. The object detection device according to claim 7, theprocesses further comprising: removing, out of the optical flows, anoptical flow in a direction along a moving direction of the camera, asan optical flow included in a background region of the object, wherein,out of regions on each of the plurality of frame images, a region inwhich a third optical flow in a direction not equivalent to that of theoptical flow removed has been obtained is determined, as a region inwhich the object is present, the third optical flow being included inthe optical flows.
 10. An object detection method comprising: acquiring,from a plurality of frame images arranged in a time series in an imagesequence successively captured by a camera, a pair of same frame imagesfor each of the plurality of frame images and inverting one of the sameframe images, generating image pyramids each of which includes imagesobtained by gradually reducing a corresponding one of the same frameimages of the pair, extracting, from an image of a salient partextracted from each of the images in each of the image pyramids, a groupof pixels each of whose pixel value is larger than a correspondingthreshold value, the threshold value being set for each of the pixelsdepending on brightness information in a vicinity of a target pixel,generating a map by integrating the extracted group of pixels,binarizing and then integrating the map generated for each of the imagesin each of the image pyramids which correspond to the same frame imagesof the pair, and thereby generating, for each of the plurality of frameimages, an image feature map representing an estimated distribution ofobject likelihood on a corresponding one of the plurality of frameimages; calculating optical flows between the plurality of frame imagescaptured successively by the camera; calculating reliability of theoptical flows calculated; and performing object detection based on atleast one of the optical flows and object detection based on the imagefeature map depending on the reliability calculated.